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A ' of wSaTi'I NEwf RIPT ' 0N ° F Y ° UR ,NVENT ' 0N ' PARTICULARLY POINTING OUT WHAT IS BELIEVED TO BE NOVEL (THE "HEART 

Bayesian networks (BN), known also as belief networks, models are graphical probabilistic models that result 
from combining graph and probability theories. They are used to model decision support problems The BN 
mode s can be created using information obtained from experts, from design documentation, and from data Thev 
can also be learned entirely from data. We describe a method and software for evaluation of BN models for" 
decision support. The method applies to all BNs independent of the way in which they were created It provides a 
systematic approach to evaluating the performance of BN. In our examples we will apply our method to BN for 
diagnostics, however the method is applicable to BN models used for any decision support problem. 

Before BN models can be used as decision support aids, they have to be extensively evaluated A typical 
evaluation relies on comparing the answers suggested by the BN models with those expected by the experts 
The evaluation is limited to a relatively small number of decision cases, for which the experts know the correct 
answer. We have developed a systematic approach to evaluation of BN and implemented it in a software tool 
The tool takes in a BN model and produces graphs characterizing the model performance. We have also 
developed a method for interpreting the graphs to identify the parts of the BN models, which are responsible for 
inadequate performance. Our technique also provides a way to analyze the domain being modeled to discover 
how well the domain lends itself to accurate decision making. 

We are not aware any other systematic method and software that addresses the problem of either Bayesian 
model or domain evaluation. 

B. EXPLAIN THE PURPOSE AND ADVANTAGES OF YOUR INVENTION. (WHAT WILL THE INVENTION DO BETTER THAN DONE PREVIOUSLY?) 

A conventional evaluation of BN models is based on a limited ad hoc testing. First, a set of cases is identified for 
which a correct decision is known. The cases may come from the data or from the expert. Then, the BN is 
queried for decision recommendations based on the cases. The quality of the BN model is determined on the 
basis of the recommendations produced by the model for the cases. The number of the cases is usually very 
limited and their selection is driven by their availability rather than proper coverage of the decision domain. The 
conventional evaluation is almost always incomplete and therefore unreliable. 

Our approach provides for a complete and informative evaluation of the BN for decision support. It is a 
completely automated and exhaustive evaluation. It informs the user about the expected performance of the 
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model in decision support tasks. Moreover, it points to the parts of the model which are responsible for errors in 
suggested decisions and helps in identifying changes of the model that could improve its performance. 

Our approach to BN evaluation shortens the time from design to practical application of decision support tools It 
also provides a solid basis for estimating the performance of the tools before they are released for use. 



8. 
C 



SUMMARY OF THE INVENTION (Continued) 

IDENTIFY THE COMPANY OR OWNER PROGRAM OR PRODUCT LINE TO WHICH THE INVENTION APPLIES AND THF FVPFrTFn \/ai , .c -rr, 
THE PROGRAM OR PRODUCT LINE. ALSO IDENTIFY POTENTIAL COMMERCIAL AP^TON OF THIS f N VENTON IF ANY ° 

The invention will be very beneficial to all the divisions that at present use decision support tools based on BN 
We are aware of use of such at GM Electromotive Division, Boeing Phantom Works, Boeing Satellite Systems' 
We know of plans to use BN models at GM Service, Boeing Commercial Airplane Divisions and Boeing Defense 
Divisions. BN models have been also used in several programs of US Government (e.g. Darpa Knowledge Data 

DdSGS, GtCy 

The software tool, which implements the evaluation algorithm, could be licensed to other companies. 

D * K°N^ UP ° N ° R ° ,SPLACED BY Y ° UR ' NVENTI0N - AND S TATE IN DETAIL, IF 

D. Heckerman, J.S. Breese, K. Rommlese "Decision-Theoretic Troubleshooting," Communications of ACM 
March 1995, Vol. 38, No. 3, pp. 49-57 

The paper describes planning of test and repair sequences for cost-optimal troubleshooting. The systems 
undergoing troubleshooting are modeled using BN. The focus of the paper is on finding the ordering of test and 
repair steps that results in minimal cost of troubleshooting. The common aspect of our invention and the work 
described in the paper is a use of Monte Carlo methods. They are applied in the paper to generate test examples 
from the BN. The examples are the basis of comparison of the author's planning method and other methods 
known in the literature. The Monte Carlo methods are not used to evaluate the BN model. 

D. Heckerman, D, Geiger, D.M. Chickering "Generating Improved Belief Networks" US Patent 5 802 256 
September 1, 1998 

The patent describes a method for creating BN models for decision support problems from expert knowledge and 
from data. The contribution of the authors is in integrating the two sources of information to obtain a model of 
better performance than that originating from data or expert knowledge only. The BN is created using a software 
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tool referred to as network generator. The patent does not discuss the evaluation of the BN obtained from the 
generator. 

E. IF PRIOR ART EXISTS, EXPLAIN WHY YOUR INVENTION IS NOT OBVIOUS IN LIGHT OF THE PRIOR ART CONSIDER SUCH FAPTncx? a? 
UNEXPECTED RESULTS, COMMERCIAL SUCCESS OF THE INVENTION, A LONG-FELT NEED THAT IS ! ^tSiED w™sS2S5S5 
FAILURE OF OTHERS WHO HAVE TRIED TO MAKE THIS INVENTION OR SATISFY THE NEED O^t^of^S^X^SSiFtt^ 

fSSSSyoptS .EtS AND SKEPT,C,SM BV TH0SE EXPERT IN THE techn'cal fXof G tSe IN^N^ON^OU^THE 

The invention addresses a very important need faced by all those who use BN in real-life decision support 
problems. The BN models designed for critical decision support problems, e.g. diagnostics, are very complex and 
need to be very carefully evaluated before they can be used in practice. The only feasible way to accomplish it is 
to use some automated evaluation method, which covers all the parts of the model and all the most probable 
decision cases. Our invention proposes such a method. We are not aware of any other method and software that 
can accomplish it. 



The two examples of prior art mentioned in Section D come closest to our invention, but they do not address the 
very problem of model evaluation. The "Decision-Theoretic Troubleshooting" paper mentions briefly an algorithm 
for comparison of planning methods for troubleshooting. Only an outline of the algorithm is presented in the 
paper. The algorithm belongs to a class of Monte Carlo methods. In our evaluation method we also use an 
algorithm of the Monte Carlo class to implement a part of our processing. There are some similarities between 
the algorithms, but there are also significant differences. A detailed comparison is however impossible because 
of lack of detailed algorithm description in the paper. 

The "Generating Improved Belief Networks" patent describes a method for creation of BN. The method is claimed 
to produce better networks than ad hoc methods. The expected improvements are justified by the construction 
method. The patent does not describe a separate evaluation step. Our invention can be used for evaluation of 
the networks created using the method from the patent. 

While the detailed implementation of our invention is complex, the concept can be copied, hence the need to 
protect it with a patent. 



9. DETAILED DESCRIPTION 



DESCRIBE YOUR INVENTION IN DETAIL, EXPLAINING THE STRUCTURE OF THE APPARATUS OR DEVICE, INCLUDING MATERIALS 
USED. SIZES AND DIMENSIONS AND HOW COMPONENTS ARE CONNECTED AND EXPLAINING THE METHOD OF PERFORMING THE 
INVENTION, INCLUDING EACH OF THE STEPS NEEDED TO COMPLETE THE METHOD. MULTIPLE EMBODIMENTS OF THE INVENTION 
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1X2^^^^ N^ A E N S 0 S N A E RY EMB0D,MENT * S D,SCLOSED ' ,DENT ' FY WH ' CH ' S ™ PREFERRED 

A. BE SURE THAT EACH SHEET IS DATED, AND SIGNED BY EACH INVENTOR AND TWO WITNESSES 

support means only ,ha. some of the BN nodes are designated as 25 ZZ£2S£L 

Knowing the probab.l.ty, the user will decide which components to repair. 

our software assumes BN models created according to the Hugin, Netica or GeNle format ' 

Let us consider an example of a diagnostic problem - simplified car diagnosis. We have seven car comoonent 
S wpp'th T 9 ^ 0bse ^ at ! ons - Let us assu ™ that we have created somehow a BN capLrin the dSencies 
between the failures and observations, figure 1. The failure nodes are depicted as blue nodes T he obs^ations 
as ye low nodes. There are also two white nodes representing auxiliary nodes, which are used or the sate of 

rnptti ' ty ' T t he H nkS b8 i Ween the n ° deS Signify the ^usa. dependencies between componenfa of the model 
n general, ourtechn.que does not require a causal model, but if causal links are not known th^Sr^rSe 
requires knowledge of a total temporal ordering of the variables. In the case of a causal model le <Mte£Za\ 
ordering can be found by performing a topological sort of the network (i.e., ordering the nodes such that f Tode A 

AH a the a n e ol°, E£ 2 th6n A COm ? S b6f0re B " the ° rderin9) - The figUre sk ° WS 4 fteSSi^SS mode, 
fhe remahing nodes P3rameters in form of Probabilities: prior for root nodes and conditional for all 

HRL PROPRIETARY AND SENSITIVE (when comnlotpri* 

LEGEND SHALL BE APPLIED TO ALL DOCUMENTS CONTAINING THIS INFORMATION. S 



SIGNATURE OF INVENTOR 



SIGNATURE OF INVENTOR 



SIGNATURE OF INVENTOR 



SIGNATURE OF INVENTOR 



SIGNATURE OF INVENTOR 



READ AND UNDERSTOOD BY: 



INVENTION DOCKET NO. 



WITNESS NAME (TYPE) 


SIGNATURE DATE ~™ " 




(EACH PAGE UPON WHICH INFORMATION IS ENTERED SHOULD BE SIGNED AND WITNESSED) 




_ I SIGNATURE DATE " 

©2001 HRL Laboratories. LLC. All right* reset 


T*d. Rev: 042501. 



SEND COMPLETED DISCLOSURE DIRECT To, 
TECHNOLOGY INFORMATION OFFICE 
ATTN: Joy Pavlow, MS RL55 
HRL LABORATORIES, LLC 
Telephone No - 5734 



HRL PROPRIETARY AND SENSITrvtE 

(when completed) 

INVENTION DISCLOSURE 

THIS INVENTION DISCLOSURE IS MADE PURSUANT TO 
MY / OUR OBLIGATIONS TO HRL LABORATORIES, LLC 




LABORATORIES 



8 of 17 



£fn™I? nti0 £ '! ! r th ° d f0r evaluation of tne BN model and the decision domain. The method provides 
information that helps in answering the following two questions- provides 

iZSZ^Vu the ' f 3 d ! a 9 nostic ^sistant?" This question can be further broken down into two sub- 
how T S \t \ How c, ° se| V does th * model reflect reality?" and (b) "Given that the model perfect reflecte realitv 
how does the domain being modeled lend itself to correct diagnosis'?" peneciiy reflects reality, 

Sn'S ™ 6e s'W a ™ {Q ™ ar « responsible for ambiguous or incorrect diagnostic suggestions'?" This question 
can also be broken down into two components: (a) "Which nodes/parameters are being incorrectly mSeted? 

fhe &SS^ Zt£T* y ref ' eCtS reaHty ' WhiCh V9riab,eS in "» - — noWSKSiK^ 

The evaluation is implemented using an algorithm, which has three basic steps- 

• Failure propagation 

• Diagnosis 

• Visualization 

In failure propagation step we perform the following computation steps: 

• select one or more specific failures 

• in the BN we set the states of nodes representing the failures to defective 

• set the states of the remaining failure nodes that are the root nodes of the BN to the state "non defective" 

• determine the state of the remaining nodes using Monte Carlo simulation 

• find the next node in the list of temporally ordered nodes 

• Using BN inference, calculate the posterior distribution of that node given the evidence so far 

• determine the state of the node by Monte Carlo sampling of its posterior distribution 

• stop when states of all nodes have been determined 

The failure propagation step is followed by diagnosis step: 

• assume the states of all the observation nodes to be those determined in the failure propagation step 

. compute posterior probability for all the failure nodes (not only the nodes selected as "defective" in the failure 
propagation step) given the states of the observation nodes 

The first two steps of the algorithm are performed many times for each specific selection of the failures Thev 
amount to selecting failures, obtaining a set of likely observations resulting from the failures and then diagnosing 
the failures as if only the observations were known. The failures may be selected systematically e.g each failure 
node separately, then all pairs of failure nodes etc. or randomly-according to the probability of failure occurrence 

HRL PROPRIETARY AND SENSITIVE (when completed) 

LEGEND SHALL BE APPLIED TO ALL DOCUMENTS CONTAINING THIS INFORMATION. ,N ° R IN PART ' THIS 



SIGNATURE OF INVENTOR DATE 




INVENTION DOCKET NO. 




SIGNATURE OF INVENTOR DATE 


SIGNATURE OF INVENTOR DATE 


SIGNATURE OF INVENTOR DATE 






SIGNATURE OF INVENTOR DATE 


SIGNATURE OF INVENTOR DATE 


SIGNATURE OF INVENTOR DATE 


SIGNATURE OF INVENTOR DATE 



READ AND UNDERSTOOD BY: 



WITNESS NAME (TYPE) 


SIGNATURE DATE 






(EACH PAGE UPON WHICH INFORMATION IS ENTERED 


SHOULD BE SIGNED AND WITNESSED) 




© 


I SIGNATURE DATE IZIlJ 

2001 HRL Laboratories. LLC. All rights reserved. Rev: 042501 



SEND COMPLETED DISCLOSURE DIRECT Tc 
TECHNOLOGY INFORMATION OFFICE 
ATTN: Joy Pavlow, MS RL55 
HRL LABORATORIES, LLC 
Telephone No - 5734 



HRL PROPRIETARY AND SENSIT> ^ 

(when completed) 

INVENTION DISCLOSURE 

THIS INVENTION DISCLOSURE IS MADE PURSUANT TO 
MY / OUR OBLIGATIONS TO HRL LABORATORIES, LLC 




LABORATORIES 



9 of 17 



Il e w« S ! eP r ViS ^ ,i2a , ti u n "i S P erformed when a » th e computations for the first two steps are completed There 
are two outputs produced by the visualization step: wimpiewa. i nere 

• complete graph for failure probabilities 

• 2D and 3D matrices of averaged failure probabilities 

The complete graph for our example network is shown in figure 2. To generate this example we individual set 
each failure node to "defective" 100 times each. Then we generated a set of likely ctoS^^^^L 

c d l a " r f ieved the Posterior probabilities of each failure given the set of likely observations In addition we 
also set the two most likely pairs of failures (Fuel-in-tank/BatteryCharge and Fuel-in-tank/Fuel Filter) to 
defective simultaneously and generated observations from these pairs. Thus we generated a total of 900 
cases. The complete graph provides a pictorial representation of all of these cases: each point on the x-axis of 
£ t S h! raP J CO ' re f f ponds Xo f .f in 9 |e case ' and V-axis denotes the posterior value for each failure or failure/pair 
in the network. If a given failure was part of the set of nodes that were set to "defective" in the simulation then its 
posterior is shown as a positive (>0) value; whereas, if the failure was not in the faulty set, then it's posterior is 
shown as a negative value. The cases are ordered from the left of the graph to the right so that the cases for the 
nodes that are most likely to be defective come first followed by the less likely cases. The gray line indicates the 
prior probability of each failure or failure pair (scaled as a proportion of the largest prior). 

The graph gives one a complete view of what happens when various failures or sets of failures are present A 
quick scan of the bottom half of the graph tells us which nodes can be implicated when a particular failure is 
present. It also gives us specific information about the possible discrete levels that each failure's posterior 
probability can take, assisting us in deciding about when a failure should be deemed to be present. 

In our automobile diagnosis example, we can see immediately that the failures "Fuel Filter" and "Fuel Pump" 
both very frequently implicate each other, and both occasionally implicate the "Solenoid", which in turn 
occasionally implicates each of them. These symmetrical implications are due to the fact that each of these 
failures have similar observations in the model. All of them can cause the engine to stop working, and "Fuel 
Filter" and "Fuel Pump" both have an impact on whether fuel gets into the carburetor. One might conclude, 
based on this graph, that even if our model perfectly reflects reality, we will not be able to distinguish between 
these three failures unless we add some observations or tests that help us separate them. 

One can also see, by looking at the complete graph, that while the "Cable Connections" failure strongly 
implicates "Battery Charge Level" failure, the converse is not true. This happens because, while again, both of 
these components impact the model in similar ways, a drained battery is much more likely to occur than a loose 
cable connection. Thus, when a cable connection is faulty, we immediately assume the battery is dead, but not 
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the converse From this information, one might conclude that we need to add to our model a strona test to 
distinguish between these two components e.g. the voltage of the battery. 9 

JU?h 2 J2 matnX ^ or ° ur + f ample 1 network is shown in fi 9ure 3. The 2-D matrix is another, more compact view of 
2 to h f n t era e l by thes, ? ulation - The ***» of this figure denotes the failures or pairs of failures tha were 
thit ?hJ 1 5? 6 ' V "T dePOteS the 3Verage p0sterior P robabilit y generated for each failure or pa gWen 
nrohil m < ° r Pa ' r ° n th6 /; aXiS W8S Set t0 " defective "- The top row of the matrix represents the prior 
probabilities of occurrence of the failures or pairs. The values are indicated by color, ranging from white (i e 

rTmntt e) h y H e ' Ti° red + (Le ' the NgheSt Value) - The 2 -° matrix 9 ives us similar informal as te 

! t 9r n P ' J d °® S ,' t amuch more concise representation. Here a quick scan of the off-diagonal 
S?p 2£5 h S Z 3 fa H S r fa '^ iS lik6ly - A SC8n ° f the dia 9° nal element * indicate * how well a true fa lure 
hSh^Lic f d t , lf ° Ur ?° de 3nd the d0main were perfect| y able t0 distinguish between the failures, then the 
diagonals for aH the single failures might be around 100% and the off-diagonals might be around 0% If the pairs 

7n Z P r^ d 'f "9 u ; shable - th f" in additi on to the diagonal value being 100%, the off diagonals cor espond no 
to the components that make up the pairs will also be implicated near the 1 00% level. wiu.ng 

In general, we are more concerned about detecting failures that are very likely to be present than those that are 
unlikely. To reflect this fact, we order the axes of the 2D matrix according to the value of the diagonal elements 
so the failures or pairs that are most likely to be correctly identified as "defective" will appear on the left/top of the 
graph. Thus a quick scan of the top row (the priors row) of the matrix allows you to identify problem targets that 
have a high probability of being faulty by looking for bright colors near the right side of the matrix In our 
example, nodes 7 and 8 ("Fuel Filter" and "Fuel Pump", respectively) have fairly high priors but their diagonal 
elements lie to the right. Also in our example, we can see the symmetric implications of "Fuel Filter" and "Fuel 
Pump", and we can see clearly the asymmetric implication of "3. Battery" by "6. Connections". Even the relatively 
minor implications are evident in this graph, for example, the cross-implication of "7. Fuel Filter" "8 Fuel Pum D " 
and 4. Solenoid". K 



The conciseness of this representation has the drawback of presenting only average values so it may not be 
possible to distinguish between a component that implicates another component many times at a low level (a 
good situation) versus one that implicates another component fewer times at a high level (less desirable 
situation). This information must be retrieved from the complete graph. 

The 3D matrix for our example network is shown in figure 4. The 3D matrix is very similar to the 2D matrix only 
instead of viewing the values using color scale, we present a full-perspective 3-D map of the data This 
representation has many of the advantages of the 2D map, but also allows us to get a better feeling for the 
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several angles to see around walls or spikes that might be present in the data. 

The software implementation of this method is a Windows executable program written in object-oriented C++ 
H°u Jn/^Tp? ram M ta t keS ;y ° BN T^ 6 ' fi,e ° f *» dedsion domain - The ««« can be in ds° ( U S netfl e 

X&ts^^ zr s the three 9raphicai repr — ns ° f the - dei ™~ 

[1] Finn V. Jensen "Bayesian Networks and Decision Graphs," Springer Verlag, New York. 2001 

B. ATTACH COPIES OF DRAWINGS OR DETAILED REPORTS HELPFUL IN UNDERSTANDING HOW YOUR INVENTION WORKS. 




Figure 1. Bayesian Network for Diagnosis of Car Problems. 
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Figure 2. Complete Graph of Failure Diagnoses for BN from Figure 1. 
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Figure 3. 2-D Matrix of Failure Diagnosis Averages for BN from Figure 1. 
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Figure 4. 3-D Matrix of Failure Diagnosis Averages for Bayesian Network from Figure 1. 
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Figure 5. 2-D Matrix for a Large Diagnostic Model #1 - Good Diagnostic Performance 
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Figure 6, 2-D Matrix for a Large Diagnostic model #2 - Poor Diagnostic Performance 
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C ' ADVANTAGES L^TED^N ABOVE."^^^' ™ E TEST RESULTS WH ' CH CONFlRM ™E ^NCT.ONS AND 

t?nnhTf^ 0d t WaS l eS ! 6d ° n f 6Veral Sma " example networks and on *k larger networks used in real-life 
troubleshooting of a large electromechanical system. One of the large models (#1) contained 169 nodes with 47 
failure nodes. The other model (#2) contained 98 nodes with 36 failure nodes. The evaluation of these networks 

^Kntrm l7pC° ^ 1 °° ^ ^ 2 ° for #1 3nd ab ° Ut 15 minuteS for 

#1 is an example of a nearly perfect diagnostic model, figure 5. The falsely implicated failures (off-diagonal 
rSirm h ^f average probability much smaller then the true failures (diagonal elements). This evaluation was 
confirmed by follow-up studies showing that a diagnostic support tool based on #1 model correctly classified 
records about 95% of the time. ' 

On the other hand, the 2D matrix for #2 model shows a much more complicated picture. The first observation 
looking across the top row of priors, is that there are 4 or 5 bright colors near the far right of the graph These5 
failures are very likely to occur (in fact the top 3 failures are part of this set), and have very low values on the 
diagonal i.e. are poorly recognized as true failures. To make matters worse, the 3 failures most likely to be 
defective (N101, N288, and N381) are strongly coupled: they all implicate each other at about the same rate as 
that they are implicated themselves. It is thus apparent that there is a need for additional tests for these failures 
The test must implicate strongly the real failure and must help in separating the failure from other non defective 
components. This evaluation was verified empirically: The best diagnostic accuracy we could achieve for the 
model #2 was about 65% . 
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